Skip to content

ARROW-9266: [Python][Packaging] Enable S3 support in macOS wheels#8315

Closed
kszucs wants to merge 4 commits into
apache:masterfrom
kszucs:macos3
Closed

ARROW-9266: [Python][Packaging] Enable S3 support in macOS wheels#8315
kszucs wants to merge 4 commits into
apache:masterfrom
kszucs:macos3

Conversation

@kszucs

@kszucs kszucs commented Oct 1, 2020

Copy link
Copy Markdown
Member

Depends on #8304

@kszucs

kszucs commented Oct 1, 2020

Copy link
Copy Markdown
Member Author

@github-actions crossbow submit wheel-osx-mavericks-cp36m wheel-osx-high-sierra-cp36m

@github-actions

github-actions Bot commented Oct 1, 2020

Copy link
Copy Markdown

Revision: 430ffdf472b9420cdbb5784946e2ba0edaf2f5c8

Submitted crossbow builds: ursa-labs/crossbow @ actions-571

Task Status
wheel-osx-high-sierra-cp36m TravisCI
wheel-osx-mavericks-cp36m TravisCI

@github-actions

github-actions Bot commented Oct 1, 2020

Copy link
Copy Markdown

@kszucs

kszucs commented Oct 1, 2020

Copy link
Copy Markdown
Member Author

@github-actions crossbow submit wheel-osx-*

@github-actions

github-actions Bot commented Oct 1, 2020

Copy link
Copy Markdown

Revision: 430ffdf472b9420cdbb5784946e2ba0edaf2f5c8

Submitted crossbow builds: ursa-labs/crossbow @ actions-572

Task Status
wheel-osx-high-sierra-cp35m TravisCI
wheel-osx-high-sierra-cp36m TravisCI
wheel-osx-high-sierra-cp37m TravisCI
wheel-osx-high-sierra-cp38 TravisCI
wheel-osx-mavericks-cp35m TravisCI
wheel-osx-mavericks-cp36m TravisCI
wheel-osx-mavericks-cp37m TravisCI
wheel-osx-mavericks-cp38 TravisCI

@kszucs

kszucs commented Oct 5, 2020

Copy link
Copy Markdown
Member Author

@github-actions crossbow submit wheel-osx-*

@github-actions

github-actions Bot commented Oct 5, 2020

Copy link
Copy Markdown

Revision: 50aaa55f306fd492cb743f011c9968860dbab2d8

Submitted crossbow builds: ursa-labs/crossbow @ actions-598

Task Status
wheel-osx-high-sierra-cp35m TravisCI
wheel-osx-high-sierra-cp36m TravisCI
wheel-osx-high-sierra-cp37m TravisCI
wheel-osx-high-sierra-cp38 TravisCI
wheel-osx-mavericks-cp35m TravisCI
wheel-osx-mavericks-cp36m TravisCI
wheel-osx-mavericks-cp37m TravisCI
wheel-osx-mavericks-cp38 TravisCI

@nealrichardson

Copy link
Copy Markdown
Member

@kszucs please rebase

@kszucs

kszucs commented Oct 5, 2020

Copy link
Copy Markdown
Member Author

@github-actions crossbow submit wheel-osx-*

@github-actions

github-actions Bot commented Oct 5, 2020

Copy link
Copy Markdown

Revision: 16a9f0916e8a9c5e10e0f66a086ad54c84676d34

Submitted crossbow builds: ursa-labs/crossbow @ actions-604

Task Status
wheel-osx-high-sierra-cp35m TravisCI
wheel-osx-high-sierra-cp36m TravisCI
wheel-osx-high-sierra-cp37m TravisCI
wheel-osx-high-sierra-cp38 TravisCI
wheel-osx-mavericks-cp35m TravisCI
wheel-osx-mavericks-cp36m TravisCI
wheel-osx-mavericks-cp37m TravisCI
wheel-osx-mavericks-cp38 TravisCI

Comment thread dev/tasks/python-wheels/osx-build.sh Outdated

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you keep alphabetical order?

Comment thread dev/tasks/python-wheels/osx-build.sh Outdated

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ditto.

@kszucs

kszucs commented Oct 6, 2020

Copy link
Copy Markdown
Member Author

@github-actions crossbow submit wheel-osx-*

@github-actions

github-actions Bot commented Oct 6, 2020

Copy link
Copy Markdown

Revision: 97b5b32

Submitted crossbow builds: ursa-labs/crossbow @ actions-605

Task Status
wheel-osx-high-sierra-cp35m TravisCI
wheel-osx-high-sierra-cp36m TravisCI
wheel-osx-high-sierra-cp37m TravisCI
wheel-osx-high-sierra-cp38 TravisCI
wheel-osx-mavericks-cp35m TravisCI
wheel-osx-mavericks-cp36m TravisCI
wheel-osx-mavericks-cp37m TravisCI
wheel-osx-mavericks-cp38 TravisCI

@kszucs

kszucs commented Oct 7, 2020

Copy link
Copy Markdown
Member Author

@github-actions crossbow submit wheel-osx-*

@kszucs

kszucs commented Oct 7, 2020

Copy link
Copy Markdown
Member Author

The successful builds take between ~42-45 minutes but occasionally it exceeds the 50m build time limit. Sadly we can't utilize travis' build cache since crossbow always creates new branches and the fallback default branch won't have any cache entries.

@github-actions

github-actions Bot commented Oct 7, 2020

Copy link
Copy Markdown

Revision: c40e6e4

Submitted crossbow builds: ursa-labs/crossbow @ actions-611

Task Status
wheel-osx-high-sierra-cp35m TravisCI
wheel-osx-high-sierra-cp36m TravisCI
wheel-osx-high-sierra-cp37m TravisCI
wheel-osx-high-sierra-cp38 TravisCI
wheel-osx-mavericks-cp35m TravisCI
wheel-osx-mavericks-cp36m TravisCI
wheel-osx-mavericks-cp37m TravisCI
wheel-osx-mavericks-cp38 TravisCI

@kszucs

kszucs commented Oct 7, 2020

Copy link
Copy Markdown
Member Author

@github-actions crossbow submit wheel-osx-mavericks-cp38

@github-actions

github-actions Bot commented Oct 7, 2020

Copy link
Copy Markdown

Revision: 7bcd83e

Submitted crossbow builds: ursa-labs/crossbow @ actions-613

Task Status
wheel-osx-mavericks-cp38 TravisCI

@corleyma

corleyma commented Oct 7, 2020

Copy link
Copy Markdown
Contributor

The successful builds take between ~42-45 minutes but occasionally it exceeds the 50m build time limit. Sadly we can't utilize travis' build cache since crossbow always creates new branches and the fallback default branch won't have any cache entries.

Does this explain why some of the nightly builds seem to be intermittently missing certain macOS wheels? E.g., sometimes for example there's no cp38 wheel, only for that wheel to come back with the next nightly publish.

@nealrichardson

Copy link
Copy Markdown
Member

Sadly we can't utilize travis' build cache since crossbow always creates new branches and the fallback default branch won't have any cache entries.

We could run something on master on the crossbow repo to warm the cache.

@corleyma

corleyma commented Oct 7, 2020

Copy link
Copy Markdown
Contributor

I think @nealrichardson is right: my understanding is that travis caching will try to pull from default branch if no cache is available for current branch, so the warming strategy could work.

@kszucs

kszucs commented Oct 7, 2020

Copy link
Copy Markdown
Member Author

Sadly we can't utilize travis' build cache since crossbow always creates new branches and the fallback default branch won't have any cache entries.

We could run something on master on the crossbow repo to warm the cache.

We would need to merge the latests tasks (multiple .travis.yml files into a single one) generated by crossbow to master automatically which would be rather complicated. Crossbow also creates diverged branches from master at the moment.

We can make a jira about supporting CI caches in crossbow though I can't see a generic solution at the moment.

@kszucs

kszucs commented Oct 7, 2020

Copy link
Copy Markdown
Member Author

Does this explain why some of the nightly builds seem to be intermittently missing certain macOS wheels? E.g., sometimes for example there's no cp38 wheel, only for that wheel to come back with the next nightly publish.

Could you send me build links? I don't entirely see your question.

@kszucs kszucs left a comment

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1, merging.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants